- Connection.java: Changed loadModel(modelName) to loadModel(modelName, workerScriptPath)
- Connection.java: Removed findPythonScript() method
- LLMCallback.java: Added Javadoc for generate() method
- JMLCLLMInferenceTest.java: Updated to pass script path to loadModel()

- Connection.java: Auto-find available ports for Py4J communication
- Connection.java: Add loadModel() overload for manual port override
- Connection.java: Use destroyForcibly() with waitFor() for clean shutdown
- llm_worker.py: Accept python_port as command line argument

Move worker script from src/main/python/systemds/ to src/main/python/
to avoid shadowing Python stdlib operator module.

- Add generateWithTokenCount() returning JSON with input/output token counts
- Update generateBatchWithMetrics() to include input_tokens and output_tokens columns
- Add CUDA auto-detection with device_map=auto for multi-GPU support in llm_worker.py
- Check Python process liveness during startup instead of blind 60s timeout

- Fix duplicate accuracy computation in runner.py
- Add --model flag and error handling to run_all_benchmarks.sh
- Fix ttft_stats and timing_stats logic bugs
- Extract shared helpers into scripts/utils.py
- Add HuggingFace download fallback to all loaders
- Fix reasoning accuracy false positives with word-boundary regex
- Pin dependency versions in requirements.txt
- Clean up dead code and unify config keys across backends
- Fix README clone URL and repo structure

- Use real token counts from Ollama/vLLM APIs, omit when unavailable
- Correct TTFT and cost estimates
- Add --gpu-hour-cost and --gpu-count flags for server benchmarks

- 121 unit tests for all accuracy checkers, loaders, and metrics
- ROUGE-1/2/L scoring for summarization (replaces quality-gate heuristic)
- Concurrent request benchmarking with --concurrency flag
- GPU profiling via pynvml
- Real TTFT for MLX backend via stream_generate
- Backend factory pattern and config validation
- Proper logging across all components
- Updated configs to n_samples=50

Replace declare -A (bash 4+ only) with a case function for
default model lookup. macOS ships with bash 3.x.

- New embeddings workload using STS-Benchmark from HuggingFace
- Model rates semantic similarity between sentence pairs (0-5 scale)
- 21 new tests for score extraction, accuracy check, sample loading
- Total: 142 tests passing across 5 workloads

- Add electricity + hardware amortization cost estimation to runner
  (--power-draw-w, --electricity-rate, --hardware-cost flags)
- Fix aggregate.py cost key mismatch (api_cost_usd vs cost_total_usd)
- Add compute cost columns to CSV output and HTML report
- Update README with cost model documentation and embeddings workload

Include all 10 benchmark runs (5 OpenAI + 5 Ollama, 50 samples each)
with metrics, samples, configs, HTML report, and aggregated CSV.

- 5 workloads x 2 models on NVIDIA H100 PCIe via vLLM
- Mistral-7B-Instruct-v0.3: strong reasoning (68%), fast embeddings (129ms)
- Qwen2.5-3B-Instruct: best embeddings accuracy (90%), 75ms latency
- Compute costs reflect H100 electricity (350W) + hardware amortization
- Regenerated summary.csv and benchmark_report.html with all 20 runs

- Connection.java: Changed loadModel(modelName) to loadModel(modelName, workerScriptPath)
- Connection.java: Removed findPythonScript() method
- LLMCallback.java: Added Javadoc for generate() method
- JMLCLLMInferenceTest.java: Updated to pass script path to loadModel()

- Connection.java: Auto-find available ports for Py4J communication
- Connection.java: Add loadModel() overload for manual port override
- Connection.java: Use destroyForcibly() with waitFor() for clean shutdown
- llm_worker.py: Accept python_port as command line argument

Move worker script from src/main/python/systemds/ to src/main/python/
to avoid shadowing Python stdlib operator module.

- Add generateWithTokenCount() returning JSON with input/output token counts
- Update generateBatchWithMetrics() to include input_tokens and output_tokens columns
- Add CUDA auto-detection with device_map=auto for multi-GPU support in llm_worker.py
- Check Python process liveness during startup instead of blind 60s timeout

Integrate SystemDS as a benchmark backend using the JMLC API. All prompts
are processed through PreparedScript.generateBatchWithMetrics() which
returns results in a typed FrameBlock with per-prompt timing and token
metrics. Benchmark results for 4 workloads with distilgpt2 on H100.

Run the embeddings (semantic similarity) workload with SystemDS JMLC,
bringing SystemDS to 5 workloads matching all other backends.

Run all 5 workloads with Qwen/Qwen2.5-3B-Instruct through the SystemDS
JMLC backend, replacing the distilgpt2 toy model. This enables a direct
apples-to-apples comparison with vLLM Qwen 3B: same model, different
serving path (raw HuggingFace via JMLC vs optimized vLLM inference).

Replace distilgpt2 toy model with same models used by vLLM backends:
- SystemDS + Qwen 3B (5 workloads) vs vLLM + Qwen 3B
- SystemDS + Mistral 7B (5 workloads) vs vLLM + Mistral 7B
All runs include compute cost flags (350W, $0.30/kWh, $30k hardware).
Increase JMLC worker timeout from 60s to 300s for larger models.

# Conflicts:
#	.gitignore
#	src/test/java/org/apache/sysds/test/functions/jmlc/JMLCLLMInferenceTest.java

- Use proper imports instead of inline fully-qualified class names
- Add try-with-resources for HTTP streams to prevent resource leaks
- Add connect/read timeouts to HTTP calls
- Add lineage tracing support for llmPredict
- Add checkInvalidParameters validation in parser
- Remove leftover Py4J code from Connection/PreparedScript
- Delete LLMCallback.java
- Remove .claude/.env/meeting_notes from .gitignore
- Trim verbose docstrings

- Use proper imports instead of inline fully-qualified class names
- Add try-with-resources for HTTP streams to prevent resource leaks
- Add connect/read timeouts to HTTP calls
- Add lineage tracing support for llmPredict
- Add checkInvalidParameters validation in parser
- Remove .claude/.env/meeting_notes from .gitignore
- Trim verbose docstrings

Supports parallel HTTP calls to the inference server via
ExecutorService. Default concurrency=1 keeps sequential behavior.

# Conflicts:
#	src/main/java/org/apache/sysds/parser/ParameterizedBuiltinFunctionExpression.java
#	src/main/java/org/apache/sysds/runtime/instructions/cp/ParameterizedBuiltinCPInstruction.java

- Delete Py4J-based benchmark results (will re-run with llmPredict)
- Remove license header from test (Matthias will add)
- Clarify llm_server.py docstring

JMLC requires the LHS variable name in read() assignments to match
the input name registered in prepareScript(). Changed X/R to
prompts/results so RewriteRemovePersistentReadWrite correctly
converts persistent reads to transient reads.

Correct SystemDS concurrency scaling numbers to match actual metrics.json
data (throughput-based instead of incorrect per-prompt estimates). Update
latency table, concurrency scaling table, run_all_benchmarks.sh for
automatic c=1/c=4 runs, and regenerate HTML report.

- Remove broken base SystemDS result directories (0% accuracy, 0ms latency
  from failed earlier run)
- Remove fabricated cost per query table (benchmarks were run without
  --power-draw-w/--hardware-cost flags, all cost data was $0)
- Fix accuracy claim: c=1 matches vLLM exactly, c=4 shows minor variation
  on reasoning (64% vs 60%) and summarization (62% vs 50%) due to vLLM
  batching non-determinism
- Add SystemDS c=1 and c=4 columns to accuracy tables
- Fix report.py to show c=1 and c=4 as separate backends instead of
  merging them into one "systemds (Qwen2.5-3B)" column
- Fix floating point truncation bug in accuracy tooltip (int(50*0.58)=28,
  now uses accuracy_count from metrics.json directly)
- Replace stale "Py4J bridge cost" references with "JMLC overhead"
- Regenerate HTML report and summary CSV

…usions

Major changes:
- Restructure README: move SystemDS architecture section before results,
  add compilation pipeline files, add JMLC code example
- Add measurement methodology note: vLLM uses Python streaming HTTP while
  SystemDS uses Java non-streaming HttpURLConnection, making per-prompt
  latency not directly comparable across backends
- Rewrite conclusions to be evidence-based: llmPredict correctness proven
  by accuracy match, concurrency scaling quantified, model-vs-backend
  distinction made explicit, latency caveat explained
- Remove MLX from supported backends table (not benchmarked), mark as
  "not benchmarked" in repo structure
- Remove fabricated OpenAI cost claim ($0.02-0.03)
- Remove "All backends overview" table (redundant with other tables)
- Simplify concurrency scaling table to throughput only (remove
  misleading effective latency columns)
- Put accuracy table first (apples-to-apples metric) before latency

…and evaluation methodology

- Fix bold-pattern regex in math number extraction: allow arbitrary text
  between number and closing ** (fixes 3 false negatives in OpenAI math,
  44/50 -> 47/50)
- Re-score all 30 result sets from raw samples.jsonl (only OpenAI math changed)
- Add complete cost comparison table with all backends including OpenAI
  API cost + local compute cost
- Add cost calculation formula with hardware assumptions
- Add evaluation methodology section explaining per-workload accuracy criteria
- Add cross-backend comparisons (SystemDS vs vLLM, OpenAI vs local,
  Qwen 3B vs Mistral 7B, Ollama analysis)
- Fix PR description scope: this is the benchmark framework PR, not llmPredict
- Fix hardware claims: Ollama/OpenAI ran on MacBook, not H100
- Add model names to SystemDS column headers (SystemDS Qwen 3B c=1/c=4)
- Explain Mistral's low math results (verbose output confuses extractor)
- Regenerate HTML report

…ests, and license headers

- Extract llmPredict logic from ParameterizedBuiltinCPInstruction into
  dedicated LlmPredictCPInstruction class for better separation of concerns
- Add structured error handling: ConnectException, SocketTimeoutException,
  MalformedURLException, HTTP non-200 responses with error body readback
- Add conn.disconnect() in finally block for proper cleanup
- Add negative tests (testServerUnreachable, testInvalidUrl) with message
  assertions verifying error messages reach the user
- Add Apache license headers to llm_server.py and llm_worker.py (CI fix)
- Rewrite benchmark framework with SystemDS JMLC backend, strict HuggingFace
  dataset loaders, and run_all_benchmarks.sh orchestration script
- Fresh benchmark results: vLLM and SystemDS with Qwen2.5-3B on H100,
  5 workloads (math, reasoning, summarization, json_extraction, embeddings)

- Run OpenAI gpt-4.1-mini on all 5 workloads (math 96%, reasoning 88%,
  summarization 86%, json_extraction 61%, embeddings 88%)
- Update README with comprehensive results: OpenAI, vLLM Qwen 3B, and
  SystemDS Qwen 3B side-by-side accuracy, latency, throughput, and cost
- Regenerate summary.csv and benchmark_report.html with 15 total runs

- math: remove 'last number anywhere' and 'last sentence-ending number'
  fallbacks from extract_number_from_response (returns None if no
  explicit answer marker found)
- reasoning: remove 'last short standalone line' fallback from
  _extract_answer (returns None if no marker found)
- embeddings: reject out-of-range scores instead of clamping (6.0 now
  returns -1.0 instead of 5.0)
- summarization: remove silent fallback to unigram overlap when
  rouge-score not installed (rouge-score is a required dependency),
  remove unused _tokenize helper and re import
- openai: remove str(resp) fallback when resp.output_text fails (let
  the error propagate instead of silently returning response repr)
- Updated tests to match new strict behavior